Tipster shogun system (joint GE-CMU): MUC-4 test results and analysis

This paper reports on the joint GE-CMU Tipster SHOGUN customization effort for MUC-4 , and analyzes the results of the TST3 and TST4 runs in comparison with the GE system . INTRODUCTION We report on the joint GE-CMU results from the MUC-4 conference and provide an analysis of syste m performance. In its maiden test, the joint SHOGUN system performed very well . On the positive side, the system achieved very good overall results . On the negative side, because the system was barely ready in time for MUC-3, it was difficult to implement and test any significant modifications . RESULTS Our overall results on both TST3 and TST4 were very good in relation to other systems . Figure 1 summarizes our results on these tests . On both TST3 and TST4, the GE-CMU system was 9-10 recall points behin d the GE system and about 4 F-measure points behind . The entire difference is due to the difference between the GE TRUMP parser, which had been developed for text applications and thoroughly tested in MUC-3 , and the CMU generalized LR parser, which was developed for machine translation and has just begun to b e tested on this sort of task . In addition to these core results, the Figure 2 summarizes our performance on the adjunct test . GE GE-CMU SCORE COMPARISO N While the performance of the GE-CMU system was not as good as that of the GE system, we view thes e results in a very positive light . First, the system with the CMU parser was reasonably close to the G E system, and it was rapidly catching up at the end of the preparations for MUC-4 (The GE-CMU syste m improved about 30 recall points between the interim test in March and the final test in May) . Second , the integrated system proved that software modules and fundamental results could be shared across sites , This research was sponsored (in part) by the Defense Advanced Research Project Agency (DOD) and other governmen t agencies. The views and conclusions contained in this document are those of the authors and should not be interpreted a s representing the official policies, either expressed or implied, of the Defense Advanced Research Project Agency or the U S Government .